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Abstract 

We consider a source coding problem with a network scenario in mind, and formulate it as a 
remote vector Gaussian Wyner-Ziv problem under covariance matrix distortions. We define a notion 
of minimum for two positive-definite matrices based on which we derive an explicit formula for the 
rate-distortion function (RDF). We then study the special cases and applications of this result. We show 
that two well-studied source coding problems, i.e. remote vector Gaussian Wyner-Ziv problems with 
mean-squared error and mutual information constraints are in fact special cases of our results. Finally, 
we apply our results to a joint source coding and denoising problem. We consider a network with a 
centralized topology and a given weighted sum-rate constraint, where the received signals at the center 
are to be fused to maximize the output SNR while enforcing no linear distortion. We show that one can 
design the distortion matrices at the nodes in order to maximize the output SNR at the fusion center. 

We thereby bridge between denoising and source coding within this setup. 
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Fig. 1. Examples of wireless sensor networks (a) with a centralized topology, (b) with an ad hoc topology 


Source coding, Sensor networks. Covariance matrix distortions. Rate-distortion functions. Noise 
reduction 


1. Introduction 

It is an inherent property of sensor networks that there are several observations of the desired 
souree due to the availability of several sensors. If the observations at different nodes are 
eorrelated, it makes sense to make use of the eorrelation among the measurements to reduee the 
eommunieation eosts. This is partieularly important for wireless networks where the resourees, 
i.e. power and bandwidth are limited. In networks with a centralized topology, as shown in Fig. 
[^(a), the measurements from all the nodes are to be sent directly to a fusion center. In this case, 
one could think of the data that is already available at the fusion center (sent by other nodes) 
as side information. For an ad hoc topology with communication among the nodes, as shown in 
Fig.[^(b), the observation at the receiving node could be treated as side information. In either 
case, due to the correlation between the side information at the receiver and the data to be sent, 
the required transmission rate could be reduced using distributed source coding [|3, Q. The 
block diagram of the resulting source coding problem with side information is shown in Fig. 
1^ The vectors x, y, and z are the desired source, the observation at the encoder, and the side 
information available to the decoder, respectively. As shown in Fig.|^ an optimal estimation x 
of X is obtained from the received data and the side information at the decoder. This setup is 
commonly referred to as the Wyner-Ziv problem after the celebrated work by Wyner and Ziv 
a. Note that this setup is less general than those shown in Fig. 1, where there may be several 
sources. However, we focus on this setup for simplicity. 
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Fig. 2. Block diagram of the remote source coding problem 


In this work, we consider i.i.d. vector sources. Note that in practice, the sources are correlated 
in time. By considering i.i.d. vector sources, we allow for correlation within each vector. This 
partially models the memory in the sources. To make full use of the memory in the signals, 
one has to either avoid the i.i.d. assumption, or let the dimension of the vector sources tend to 
infinity. For our derivations, we use the i.i.d. assumption and finite dimensions, and therefore 
accept some level of suboptimality. 

The Wyner-Ziv problem first appeared as an attempt to generalize the Slepian and Wolf’s 
results in llH for lossless coding of two digital sources to the case of lossy source coding. The 
main result in (Sl was that, it is possible to encode two digital sources losslessly and separately 
with a sum-rate that asymptotically achieves the joint entropy of the two sources given that the 
decoding is performed jointly. In Q, Wyner and Ziv considered a similar problem for lossy 
source coding, where one source is to be encoded and sent to the decoder and the other one is 
available only to the decoder and serves as the side information. The results were later extended 
to continuous sources in |j6l. It was shown that in general, this setup incurs a loss compared to 
the case where the side information is available at the encoder, but if the sources are Gaussian, 
no loss occurs. 

Since in our setup the observation y is in general not the same as the desired source x, we 
refer to it as the remote Wyner-Ziv problem. In this paper, we consider the remote Wyner-Ziv 
problem with a covariance matrix distortion constraint. We derive a closed-form formula for the 
rate-distortion function (RDF), and then study the special cases and applications of this result 
to source coding problems in networks. In order to formally define the problem, we introduce 
the distortion constraints which are of interest for this work in Section [I-Bj but before that, we 
need to define our notation. 
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A. Notation 

We denote matriees and veetors by boldfaee uppercase and lowercase letters, respectively. We 
consider zero-mean stationary Gaussian sources, which generate independent vectors x G 
y G and z G as shown in Fig.|^ The ordered set of vectors (xi, X 2 , • • ■ , xat) is denoted 
by {xj}^^ for compactness. We denote the expectation operation by E[-]. The covariance and 
cross-covariance matrices are denoted by the symbol S followed by an appropriate subscript. As 
an example, for random vectors x, y and z, the conditional cross-covariance of x and y given z 
and the conditional covariance of x given y and z are denoted by Sxy|z and Sx|yz, respectively, 
and are defined as: 


Sxy|z = T; [(x - ^x|z)(y - |z] , 

Sx|yz = [(X - Mx|y,z)(x - Mx|y,z)^ IY, z] , 

where the superscript ^ denotes the transpose of a matrix, and n denotes the mean. For instance, 
/^x|y,z = z] is the conditional mean of x given y and z. 

Markov chains are denoted by two-headed arrows as in x o y -H- u, which means that if 
y is given, then x and u are independent. The information theoretic operations of differential 
entropy and mutual information are denoted hy h{-) and / (■;■)’ respectively. The trace operation is 
denoted by tr(-). The set of symmetric positive-definite matrices is denoted by §+. The statement 
A ^ B (A B) means that A — B is positive semidefinite (definite). The nxn identity matrix 
is denoted by I„. We use diag{Aj,i = to show an n x n diagonal matrix having 

the elements Ai,..., A„ on its main diagonal. We also make use of the following notations for 
briefness: 


(a)^ = max (a, 1), (a) = min (a, 1). 


B. Distortion Constraints 

Consider the remote Wyner-Ziv problem where the decoder makes an estimation of the desired 
source x G using the received message w G M"™ and the side information z G For 
the special case where the sources are scalar, the well-known mean-squared error distortion 
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constraint is in the form of < D, where D is a given target distortion, and is the 
varianee of the reeonstruetion error. With this distortion eonstraint, the sealar Gaussian remote 
Wyner-Ziv problem was treated in |E1-||91, and the rate-distortion funetion was derived in a 
rather simple form. 

For the veetor Gaussian ease, there are several options to define a quadratie distortion eon- 
straint. The most obvious one is the mean-squared error distortion eonstraint, whieh is the 
generalization of the above-mentioned distortion eonstraint for the ease of veetor sourees. It is 
defined as follows: 


tr (Sx|wz) ^ iT'xD, 

where D is a given target distortion. Another eonstraint, whieh is related to a relay network 


problem as diseussed in Seetion III-B is a mutual information eonstraint defined as: 


I (x; w|z) >Ri 

where Rj is a given eonstant. For both the mean-squared error distortion and mutual information 
eonstraints, the veetor Gaussian problem was solved in ifT^ . and turned out to have a parametrie 
form, resembling the familiar water-filling solutions for some other Gaussian rate-distortion 
problems. 

The last eonstraint we diseuss is the eovarianee matrix distortion eonstraint, whieh is defined 
as: 


Sx|wz ^ D, (1) 

where D is a given positive-definite target distortion matrix. For the veetor Gaussian Wyner- 
Ziv problem with the eovarianee matrix distortion eonstraint, the rate-distortion funetion ean be 
derived using standard teehniques for a limited set of distortion matriees (e.g. D ^ Sxiz) uni. 
However, the general ease does not appear to be manageable to solve using standard teehniques, 
and no elosed-form statement is available for the general RDF in the literature. In this paper, 
we treat this problem and study some of its applieations. We elaborate on some aspeets of this 
problem and our motivation in the next subseetion. 
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C. Motivation 

During the recent years, there has been a shift from the traditional mean-squared error dis¬ 
tortion constraint to covariance matrix distortions in the area of multiterminal source coding 

ci-iini. 

In [fT^ . a multiple description scenario is considered where L encoders send their descriptions 
of a vector Gaussian source to L individual and one joint decoders. Each decoder makes an 
estimation of the source within a given covariance matrix distortion constraint. The sum-rate of 
the multiple descriptions problem is specified in lfT2ll under certain assumptions on the distortion 
matrices. 

The Gaussian one-helper problem with the covariance matrix distortion constraint was studied 
in flu, ca. Here in addition to a main encoder which observes the direct source and sends 
a message to the decoder with rate Ri, there is a helper with noisy observations, which sends 
a message with rate R 2 to the decoder. The decoder estimates the source using the received 
messages. The rate regions for this problem for the scalar and general helpers were derived in 
Cl and [in . respectively. 

In addition to the above works, there are also several results which bound the rate-distortion 
regions of source coding problems under the covariance matrix distortion constraint CSII - IITtII . 
In the so-called CEO problem ffH . a number of agents encode and send their observations of 
a source with certain rates to a CEO. The CEO makes an estimation of the source using the 
received messages from the agents. The rate region for the vector Gaussian CEO problem under 
the covariance matrix distortion constraint was outer-bounded in IfTSl . C^ . Einally, in [fTTII . a set 
of L noisy observations of K correlated scalar Gaussian sources are encoded separately and sent 
with the rates i?i, ■ ■ ■ , Rl to a joint decoder. The decoder makes an estimation of the sources 
using the received messages. The rate-distortion regions for this problem are inner- and outer- 
bounded for three different distortion constraints including the covariance matrix distortions. 

Although the vector Gaussian remote Wyner-Ziv problem has a rather simple setup, we could 
not find any closed-form solution for the general problem under the covariance matrix distortion 
constraint in the literature. Neither does such a closed-form solution follow as a special case of 
the above-mentioned results. 

In general, with the covariance matrix distortion constraint, the matrix form of the target 
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distortion gives rise to new issues eompared to the sealar target distortions, whieh make the 
problem harder to solve. For the Wyner-Ziv problem, the eovarianee matrix of the unknown part 
of the souree at the deeoder is Sx|z- With a sealar target distortion d, the following two eases 
might arise: 

• tr(Sx|z) < n^d, for whieh the required rate is zero. 

• tr(Sx|z) > nxd, whieh means that some nonzero rate has to be spent. In this ease, with an 
optimal eoding seheme, the varianee of the reeonstruetion error at the deeoder would be 
the same as the target distortion. 

For a matrix target distortion D on the other hand, it might happen that none of the two eases 
Sx|z ^ D and Sx|z ^ D hold. In general, there are two sets of direetions in In one set of 
direetions, the distortion eonstraint is already satisfied at the deeoder using the side information z. 
In the other set of direetions, the distortion eonstraint eannot be satisfied without some help from 
the eneoder, making the minimum required rate a nonzero value. In sueh eases, the eovarianee 
matrix of the reeonstruetion error at the deeoder is not guaranteed to be equal to D. 

The argument for eonsidering the eovarianee matrix distortion eonstraints despite the above- 
mentioned issues is the generality of the results. For the remote Wyner-Ziv problem, the resulting 
rate-distortion funetion i?(D) would be a mapping from 'D = {Dg§’''|D:^ Sx|yz} to M. For a 
given problem with a eonstraint on D, there is in general a subset of V for whieh the eonstraint 
is satisfied. We ean then ehoose one member of this subset whieh minimizes the rate. As an 
example (among other examples that will be studied in this paper), we will show later on, that 
the ubiquitous ease of the RDF with the mean-squared error distortion is simply equivalent to 
i?(D) at a partieular distortion matrix D* in V. This partieular D* is the one that minimizes 
the rate R(D) subjeet to a mean-squared error eonstraint on D. 

In addition to generality, the eovarianee matrix distortion eonstraint has other advantages. 
Defining the distortion with respeet to the eovarianee matrix of the reeonstruetion error allows 
for formulating new problems. In many sensor network setups, there are eonstraints on the sum- 


rate or weighted sum-rate of the network. As we will show in Seetion III this ean be formulated 
as a eonstraint on the determinant of the distortion matriees. One ean then optimize a funetion 
(sueh as SNR) over the network subjeet to this eonstraint in order to optimally alloeate the rates 
and distortions to the sensors. 
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D. Overview of the Paper 

Section |I^ is dedicated to the vector Gaussian remote Wyner-Ziv problem under covariance 
matrix distortion constraints. We start with defining a minimum for a pair of symmetric positive- 
definite matrices based on the joint diagonalization of the matrices. We then derive some 
properties for this notion of minimum of two matrices, based on which we find an explicit formula 
for i?(D). This minimum of two matrices seems to be natural to our source coding problem, 
and in addition to the RDF itself, it appears also in the coding schemes and the reconstruction 
error at the decoder. To derive the RDF, we first lower-bound i?(D) using the properties of 
the minimum of two matrices and information-theoretic arguments. Next we upper-bound it by 
suggesting a linear coding scheme. The lower and upper bounds coincide, thus yielding the RDF. 

In Section we present examples of applications and special cases of the results obtained 
in Section 1^ We consider three applications. First, we consider a similar source coding problem 
under the mean-squared error distortion constraint, and will show that the resulting RDF is a 
special case of i?(D) for a certain choice of the distortion matrix D. 

Next, we consider a relay network scenario where in addition to the data transmitted to the 
center from a main transmitter, there is a relay which would like to help the main transmitter by 
sending its own observation to the center with a certain rate. The problem could be formulated 
as a source coding problem giving a rate-information function [flOl . We will show that similar 
to the previous case, the resulting rate-information function will be given by i?(D) for a certain 
choice of the distortion matrix D. We then show how one could implement this scenario using 
the results from Section JIl 

Finally, as the third and most elaborated example of the applications, we consider a sensor 
network with a centralized topology. The observation from each sensor is encoded and transmitted 
to the fusion center with a certain rate. We assume that the the weighted sum-rate of the network 
is limited to a given amount. At the fusion center one would like to fuse the received data in 
a manner that the output experiences no linear distortion. We will show that under the given 
weighted sum-rate constraint, one could design and allocate the distortion matrices to the sensor 
nodes in order to maximize the output signal to noise ratio at the fusion center. 

We conclude the paper in Section by discussing the main results and possibilities for future 


work. 
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II. Vector Gaussian Remote Wyner-Ziv Problem 

In this section, we solve the source coding problem in networks which was simplified to the 
block diagram in Fig. Later in Section |I^ we will show how we can use the results for 
optimizing certain functions in networks. We start with a formal definition of the source coding 
problem that was introduced in Section [ij Assume that {xj,yj,Zi}^^ is a sequence of i.i.d. 
zero-mean random vectors such that Xj G ji G and Zj G are jointly Gaussian for 
i = 1, - ■ ■ , N. The encoder observes and using an encoding function: 

0(N) . ^nyN ^ {1^... 

sends a message to the decoder. The decoder observes and receives the message from 

the encoder, based on which makes an estimation of using a decoding function: 

: {1, • • • , X ^ 

Definition 1. The rate-distortion pair {R, D) is achievable if there exists a block length N and the 
encoding and decoding functions and such that for {xj}^]^ = (0^^^({yi}^i), {zj}^J.‘ 


R > 

D ^ 


ilogA/W, 


1 

N 


N 





(Xi - 



i=l 

Definition 2. The rate-distortion region for the above problem is the closure of the set of 
achievable rate-distortion pairs {R, D). 


Definition 3. Let TZ be the rate-distortion region for the above problem. The RDF i?(D) is then 
defined as: 


i?(D) = inf R. 

(R,D)eR 


Notice that Sx|yz is the covariance matrix of the reconstruction error at the decoder when in 
addition to the side information z, the uncoded observation y of the encoder is also available at 
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the decoder. Since this is not achievable with finite rates, we must have Sx|yz “< D, otherwise 
the rate would become infinite. We thus assume that it holds. 

Suppose that the auxiliary random variable u satisfies the Markov chain u y O (x,z). 
From the results in it follows that the operational RDF defined above is equal to the following 
information RDF: 


/?(D) = min / (y; u|z) s.t. 


^x|uz D, 


( 2 ) 


u^y ^ (x,zj. 

We write x in terms of its linear estimation from y and z and the estimation error n as 
follows: 


X = Ay + Bz + n, (3) 

where n is independent of y and z, and Sn = Sx|yz- See Appendix [A| for a quick derivation of 
the matrices A and B. 

Defining y' as: 


y' = Ay, (4) 

and using Q and the Markov chain in Q, one can write: 

Sx|z = Sy/|z + Sx|yz5 
^x|uz ^y'|uz 3“ ^x|yz- 

We solve Problem Q in the rest of this section. To do so, we 
minimum for two positive-definite matrices in the next subsection. 

A. Minimum of Two Matrices 

Based on the joint diagonalization of two matrices, we define a minimum for two positive- 
definite matrices and derive some important properties of this definition. These properties will 


(5) 

( 6 ) 

first introduce a notion of 
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be crucial in the derivation of the RDF in the next two subsections. Although any joint diag- 
onalization would work for this matter, we focus on a particular case which is defined in the 
sequel. 

Consider the ordered pair (Si, S 2 ) of two nxn symmetric positive-definite matrices. Denote 
the eigenvalue decomposition of Si by: 


Si = U^AU, 

where A = diag{Aj, i = 1,..., n} and Ai > A 2 > ■ ■ ■ > A„. Consider the principal square-root 

1 1 „i _1 

S^ = U^AzU of Si and denote the eigenvalue decomposition of S^ ^S 2 S]^ ^ by: 

s’^SsS”^ = w'^rw, 

such that r = diag{ 7 j,i = and 71 >72 > > 7 n- We then define the joint 

diagonalizer of (Si,S 2 ) as follows. 

Definition 4. The joint diagonalizer V of the ordered pair (Si,S 2 ) is defined as: 

V = A^WSP. 

One can verify that: 


VSiV^ = A, 

VS 2 V^ = A', 

where the diagonal matrix A' = diag{A', i = 1,..., n} is defined as A' = AF. 

This variant of joint diagonalization is based on another form of diagonalization found in [fT9l 
Theorem 8.3.1]. Based on the above joint diagonalization, we define the minimum of (Si, S 2 ) 
as follows: 

Definition 5. The minimum of the pair (Si, S 2 ) of symmetric positive-definite matrices with the 
Joint diagonalizer V is defined as: 


min(Si,S 2 )=V Miag{min(Aj, A'), z = 1,..., njV 
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Some properties of the above definitions are summarized in the following lemmas. 

Lemma 1. For the pair (Si,S 2 ) of symmetric positive-definite matrices with the joint diago- 
nalizer V the following properties hold: 

1) |V| = 1. 

2) a; > A' > • • • > a;. 

3) min (Si, S 2 ) ^ Si, and min (Si, S 2 ) ^ S 2 . 

4) min(Si,S 2 ) = Si if and only z/Si ^ S 2 , and min(Si,S 2 ) = S 2 if and only if 

S 2 ^ Si. 

Proof: The proof follows immediately from the definitions and is thus left out. ■ 

The proof of the following lemma follows from the results in li20ll . 

Lemma 2. Consider the following optimization problem: 


a 


* 


max |S| s.t. 

seS+ 


S ^ Si, 
S ^ S 2 , 


where S, Si, S2 G S’*". We then have a* = |min (Si, S2)|. 


B. Rate Distortion Function 

The following theorem is the main result of this seetion. 


Theorem 1. The rate-distortion function for problem ^ for D ^ Sx|yz is given by: 


2 , |lXlin(D Zjxjyz? ^x|z ^x|yz) 


(7) 


Proof: We prove the theorem by deriving eoineiding upper and lower bounds on the RDF 
in Sections ITC and II-D[ respectively. ■ 


C. Lower Bound on i?(D) 

Let V be the joint diagonalizer of (Sx|z — Sx|yz, D — Sx|yz) as given in Definition^ such 

that:E] 

'We assume that Sx|z ^ ^x|yz is of full rank. The rank-deficient case can be converted to an equivalent problem with 
full-rank matrices using an appropriate transformation. See e.g. Appendix A in [131 
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V(S,|, - = A, 

V(D - Sx|yz)V^ = A'. 

Define the function -R(D) as: 

^ '°g I ■ rn 1 "?'^ ~ 

2 jniinfD Sj^lyzi ^x|z ^x|yz)| 

The following lemma is then the main result of this subsection. 

Lemma 3. The RDF i?(D) defined in (j^ is lower-bounded as i?(D) > fl(D). 

Proof: See Appendix 


( 8 ) 

(9) 


( 10 ) 


D. Upper Bound on i?(D) 

In this subsection, we upper bound i?(D) with the same function .R(D) which appeared in the 
lower bound in the previous subsection. This thus in combination with the lower-bound, gives 
the RDF in a closed form. It is important to note though that the knowledge of the RDF does not 
necessarily specify how one could encode the observations to achieve i?(D). The upper bound 
derived in this subsection is constructive, in the sense that it is based on an achievable coding 
scheme. Such a scheme should suggest a way to transform y into another variable u, such that 
for a given distortion D: 

• The required rate for delivering u to the decoder is no higher than i?(D). 

• The decoder could make an estimate x of x using u and z, such that the distortion constraint 
([T]) is satisfied. 

The role of u here is to model the quantization effect on y (or a transformation of y). It 
is thus typically composed of a linear transformation of y and an additive noise term that 
represents the quantization noise. We suggest such a scheme in the sequel. Suppose that V, 
A = diag{Ai, • • ■ , An}, and A' = diag{A}, • • ■ , A'„} are defined as in ([^ and (|^. Suppose also 
that U is the orthogonal matrix of the eigenvectors of Sx|z — Sx|yz giving the eigenmatrix A. 
The result of this subsection is then given by the following lemma. 


Lemma 4. The following Gaussian test channel achieves i?(D) given by (10): 
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u* = UAy + j/, (11) 

where A is defined in ([^, and the the coding noise v is independent of y with the covariance 
matrix given by: 


S, = UV-Miag <1 , = 1 ,..., ri, >>V-^U^ 


\ • :\ x/N’" ^ ■ (12) 

^Aj-min (Aj, A') 

Moreover, the covariance matrix of the reconstruction error at the optimal decoder is given by: 


^x|u*z ^x|yz “1“ Hllll ^Sx|z ^x|yz) D Sx|yz) • 


(13) 


Proof: See Appendix 


Note that the coding scheme given by ( [TT] ) involves two steps: 

• Linear transformation of the observation y into UAy, 

• Quantization of the resulting sequence such that the quantization noise u becomes Gaussian 
and independent of y with the covariance matrix given by ( [T^ . 

The second step can be performed using a high-dimensional dithered vector quantizer. The 
dithered quantizer guarantees that u will be independent of y, and the high dimension allows 
for the Gaussianity of u (211. 

Finally, notice that an immediate result of the above lemma is that i?(D) < i?(D), which 
establishes the desired upper bound on the RDF. 


Remark: The first and second terms at the right-hand side of (13) are related to the error due 
to the remoteness of the source and the coding artifacts, respectively. This separability resembles 
the Wolf and Ziv’s result in [E3, where it is shown that for a point-to-point remote joint source- 
channel coding problem with mean-squared error distortions, the end-to-end distortion can be 
decomposed as the sum of two terms. One term is due to the remoteness of the source and 
the other one is due to coding. Note however, that it was shown in ll23ll . that this result is not 
optimal for the multiterminal case. 
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III. Applications and Special Cases 
A. Mean-Squared Error Constraint 

For the vector Gaussian remote Wyner-Ziv problem with a scalar mean-squared error con¬ 
straint, the rate-distortion function is given by ffTOll : 


/?(/)) = mm/(y;u|z) s.t. tr (Sx|uz) <uf-)-y-H-(x, z). 

It was shown in [|T^ that R{D) is given by: 

i=l ^ ' 

where A^; i = 1,2,..., are the eigenvalues of Sx|z — Sx|yz, and A is a constant (water level) 
satisfying the following constraint: 

^min(A, Ai) =n^C>-tr(Sx|yz) . (14) 

i=\ 

We will show that this is equivalent to a special case of i?(D) for a particular choice of the 
distortion matrix D. Denote the eigenvalue decomposition of Sx|z — ^xiyz by U^diag{Aj,i = 
1 ,..., nx}U, and define: 

D* = Sx|yz + U^diag{min(A, A*), i = 1,..., 

where A is defined by ( [T4] ). It is clear that D* Sx|yz and is thus a valid distortion matrix. 
Substituting D* in (|7]) yields: 


R{T>*) 


-I CLx 

jE'oS 

i=l 

-1 Elx 


A* 


min (A, Aj) 



= R(D). 
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B. Relay Networks 

Consider a relay network where a main transmitter uses antennas to transmit the Gaussian 
signal X to the end receiver which has its own noisy observation z of x using antennas. There 
is also a relay that makes the noisy observation y using its riy antennas and transmits to the end 
receiver with a given maximum rate. It is desired to find the minimum rate at which the system 
can reliably transmit. Assuming that the statistics and channel state information for x are known 
to the relay and receiver, it was shown in ifTOl that this problem is equivalent to the following 
optimization problem: 


= min/ (y; u|z) s.t. /(x; u|z) >i?/, u<H-y-f-)-(x, z), 


for a given Rj, and the resulting rate-mutual information function R{Ri) for 0 < Ri < 


I log 


’-‘x\yz \ 


is given by: 


RiRi) = 2 

i=l 


i^i 


1 /ij 

1-7 


(1 


-V 


where /ij, i = 1 ,..., are the eigenvalues of — S^|^Sx|yzS 
the following: 


2 

x|z 




(15) 

and 7 G [0,1) satisfies 


1 

2 


log 



Ri. 


(16) 


See IfTOl and the references therein for more details on the problem setup and formulation. We 


will first show that R {Rj) is a special case of i?(D) for a particular choice of the distortion 
matrix D, and then using the results of Section we will show how to design a coding scheme 
that achieves R{Rj). 

_1 _1 

Suppose that the eigenvalue decomposition of — S^|^Sx|yzS^|^ is given by: 


I 


rix 




W'^diag{/ii, i = n^}W. 


(17) 


fin (TOl, fii are defined as the nonzero eigenvalues of Note however that for any matrix Q the 

zero-excluded multispectrum is the same for QQ^ and Q^Q. One can then see that fit are also the eigenvalues of 


ASy|zUsing i4i and i5i, this simplifies to — S 


I SxivzS 7. 
x z X z 











17 


Define: 


D* = S^I^W^diag<|min( 1, 




1-7 


,7 = 1 ,.. .,nAWTi 


x|z’ 


(18) 


_1 _1 

where 7 is given by (16). From the faets that 7 < 1 and S^|^Sx|yzS^|^ ^ and using (17) 
and (18), it follows that S^.^SxiyzS^I^, or equivalently D* ^ Sx|yz- Therefore, 


D* is a valid distortion matrix. Substituting D* in (0, noting that D* ^ Sx|z> and using property 
4 in Lemma [1} we have: 




^log 


i|og 


llog 


-'x|z 


■ix|yz| 


min(D Sx|yz) 5 jx|z Sx|yz) 


|51x|z 5jx|yz| 


|D* -Sx|yz 

-s 


1 


x|z ^5c|yz^x|z 


_ 1 

S. 5 


S ,^D*S ^ 


■'x|z'^ 


■'x|z 




x|z^x|yz^x|z 


1 

s, ^ 


-j 


/ fXj l-7 \ 
U-/^* 7 / 


R{Rj). 


(19) 

( 20 ) 


where ( [20] ) follows from some straightforward manipulation of ( [T5| ) and eomparing the result 
with ( fl^ . Recall that for any valid distortion matrix D, the coding scheme ([TT]) achieves i?(D). 


Since D* given by (18) is a valid distortion matrix, to implement a coding scheme that yields 
R{Ri) = i?(D*), one could set D = D* and use ( [TT] ). 


C. Noise Reduction in Sensor Networks 

A simple diagram of a sensor array as well as a centralized network of wireless sensors is 
shown in Fig. [^ In both cases, there is a central processing unit which makes an estimation of 
the desired source using the received signals. For the sensor array, the signals which are fed 
to the processing unit are impaired versions of the desired source affected by the environment. 
One can then design the processing unit so as to minimize the noise at the output. For the 
sensor network, the sensor signals which are already impaired by the environment effects, have 
to be transmitted to the center with limited rates via wireless links. This incurs extra noise terms 
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(a) (b) 


Fig. 3. (a) A sensor array, (b) a centralized wireless sensor network. 


which are due to digital transmission (eoding noise). Although this extra noise term degrades 
the performanee, its impaet ean be alleviated by the faet that the eoding noise ean be eontrolled 
by the designer. In other words, in addition to the proeessing unit, one ean make use of the 
distortion matriees as design parameters in order to minimize the output noise at the proeessing 
eenter. Let us elaborate on this idea in the sequel. 

Consider an array of N sensors, where sensor i, i = 1, - ■ ■ , N observes a noisy version yi{k) 
of a Gaussian souree Xi{k) given by: 


yi{k) = Xi{k) + ni{k), 


( 21 ) 


where k is the time index, and ni{k) is the additive noise which is assumed to be Gaussian 
and independent of the souree Xi{k) and the noise terms at the other nodes. Define y* G as 
Yi = [yi{k — n + 1) ■ ■ ■ yi{k)Y (and similarly for x* and rij). One eould then rewrite (21) in 


the vector form as: 
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Yi = Xj + iij, (22) 

where we drop the dependeney on k for simplieity of notation. The sourees x*;* = 1, • • • 
are related to a desired source x^ as: 


Xj = Wf Xd, i = 1, 


(23) 


where W*, 


(23) as: 


1, • • • are n x n matrices. Defining W as W = [Wi ... W^v], we rewrite 


X = W'^Xrf. 

where x = [x^ ... x^]^ (and we also define y and n similarly). At the processing unit, as shown 
in Fig.[^(a), the output z is produced using the following filtering operation: 

N 

z = ^ HiYi = Hy = Hx + Hn, 

i=l 

where H = [Hi ... Hat]. The objective is to maximize the output SNR of the filter H while 
enforcing only additive distortion to the reconstruction (no linear distortion). A typical example 
of this scenario with no linear distortion is in microphone arrays, where the linear distortion 
would cause undesired artefacts in the resulting audio or speech signal [[24l . Another example 
of applications is in Networked Control Systems, where the feedback data is transmitted via a 
communication network. In certain cases, because of the implementation constraints the source 
coding scheme used to compress the feedback information should incur no linear distortion [|25l . 
To formulate the problem, notice that the output error of the filter H is given by: 

e = z - Xd = (HW^ - I) Xrf + Hn = Ox + e^, 

where is the linear distortion of the signal and On is the additive distortion. To minimize the 
additive distortion while suppressing the linear distortion, one could choose H = Unid, where 

Unid = axgm.mti{E[eneJ']) s.t. HW^ = I. 

H 


( 24 ) 
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There is a elosed form solution for (24) given by 


= (WS-^W^) 'WS~\ 
which gives the following SNR at the output: 


(25) 


SNR{Hnid) 


tr(SxJ 

tr{(WS-iW^)-^}‘ 


(26) 


Now suppose that instead of N array sensors, we have a network of N wireless sensors as 
shown in Fig.[^(b). Node i encodes its observation y* and transmits it with the rate -R(Dj) for a 
target distortion Dj. This RDF is equivalent to (|7]) for the special case where Ux = riy = Hz = n, 
X = y and z = 0. The result is given by: 






■'yii 


min(Di,SyJ 


We further assume that Dj From property 4 in Lemma it then follows that: 


fl(D.) = I log (27) 

Note that we have assumed independent decoding of the messages received from the sensors. 
This is in general suboptimal. An optimal strategy requires a joint decoding similar to the CEO 
problem ifTSll . which would be difficult to solve analytically and also to implement. 

From the definition of the distortion constraint ([T]) and the Gaussianity of the achievable coding 
scheme (see also the proof of Lemma |^, it follows that the reconstruction y^ at the decoder 
satisfies y* — y ■ = with i^i being independent of y- and = Dj. Equivalently, one could 
write: 


y' = A,yi + 

where and y* are independent, and: 


A. = (Sy. - D.)S-‘ 

Sjy/ — A^D^. 
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We apply the invertible map A ■ ^ to y' to obtain the following equivalent (but preferred) version 
of the reeonstruetion: 


Yi = A- = Yi + 

with: 


S 


I'i 



Combining (22) and (28) we have: 


(28) 


(29) 


y* = Xj + rij + i/j = Xj + Vj, 


(30) 


where Vj = n* + r/j. Reeall that the objeetive is to maximize the SNR at the output of the 
proeessing unit using the reeeived noisy signals fi, while ineurring no linear distortion to the 


output signal. Comparing (30) to (22), it is elear that this problem is similar to the ease of the 


sensor array. One ean thus apply the same filter as (25) to yi;i = 1, ■ ■ ■ ,N to maximize the 


output SNR. To do so, we rewrite (30) as: 


y = x +V, 


(31) 


where y = [yf ... and x and v are defined similarly. The output SNR similar to (26) is 

then given by: 


SNR(H„;(i, Di, ..., Dtv) — 

__. (32) 

tr I [W (Sv(Di,..., D^))-' W^] } 

Notiee that due to the wireless link between the sensors and the eenter and the limited 
transmission rate, in addition to the additive noise n* in the measurements, there is also a 
reeonstruetion error Ui due to quantization and eoding. Therefore, the eovarianee matrix of the 
noise term v in ( [3T] ) depends on the set of distortions as emphasized by the notation in ( [^ . 
While the quantization error is inevitable, one eould shape the eovarianee matrix Sv(Di,..., Dat) 


in (32) by manipulating the distortions Di, ...jDat in order to maximize the output SNR. 
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To be more specific, suppose that it is required that the weighted sum-rate of the network 
^^1 Q!ji?(Di) does not exceed a given amount R for a given set of weights ai,...,Q!Ar. The 
problem can then be formulated as follows: 


N 


max SNR(H„zrf,Di,...,DAr) s.t. 
Di,...,Djv 


'^aiR{'Di) = R. 


(33) 


Z=1 


The weights ai,a n could be chosen for example to equalize the power consumption at 
the sensors, or to give the network sum-rate by being set equal to each other, or to satisfy any 
other criterion specified by the user. Without loss of generality, we assume that: 


N 


= 1 - 


(34) 


i=l 


It is worth mentioning that this problem includes the allocation of the rates to the nodes, but 
is not limited to it, since rate allocation is equivalent to the specification of the determinant of 


the distortion matrices Dj according to (27), while in (33) the whole matrices Di, ...jDat must 
be chosen optimally to maximize the output SNR. This possibility is a result of having the target 
distortions in the matrix form. 


Problem (33) is equivalent to the following problem: 


min tr{[W(Sv(Di,...,DA,))-'wT 

N 

s.t. ^^aii?(Dj) = R. 


(35) 


2 = 1 


Substituting (27) in the constraint in (35), noting that = Sn- -f and using (29), we 


rewrite (35) as: 


N 


N 

S.t. i[\B,r=P, 


-1 


1 -1 


Wj 


i=l 


(36) 


where /? is defined as: 
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N 




(37) 


2=1 


For simplicity, we assume that W* in (23) are invertible. Notiee that even if the relationship 
between Xj and is not invertible, one eould find an invertible matrix Wj whieh minimizes 
for example the mean-squared error between x* and Wf x^. In this ease, (23) would be an 
approximation (See [l24]l for more details). However, the approximation error eould be added to 
the additive noise iij in (22). One eould thus assume that = Wf + rij always holds with 
an invertible W, . 


We define Z, and C, as: 


Z. = W,E-‘ (S-,' + Dr' - S-'Wf, (38) 

C. = WiE-' (S-' - E-') E-'Wf. (39) 

The following set of equations then follow from the KKT conditions for Problem ( |3^ (See 
Appendix [P] for the derivation): 


= Zj — ZjCj ^Zj, 

N N 

A = y^w,s-'wf-y;z, 


2=1 


2=1 


N 


n |s-'wrz-'w,s-' - S-' + = 13. 


2 = 1 


(40) 

(41) 

(42) 


In general, it is not easy to solve (40)-(42) analytieally to find the unknowns A, A, and 
Zj;z = 1, • • • ,N. Instead, one eould proeeed by using numerieal methods. In the rest of this 
subsection, we eonsider special cases of this problem where further analysis is possible and leads 
to interesting results. 


1) High-Rate Regime: We assume that (40) eould be approximated as: 


ofjAA^ Zj. 


(43) 
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From ([38]) and (39), this means that the distortion matriees Dj in (38) have small eigenvalues, 


or equivalently, has large eigenvalues. This holds, if the rates i?(Dj) are high enough. We 
will show later, that in faet in order for the results to hold, the rates need not be very high, 
and depending on the setup, even for relatively low rates the results will be valid. The high-rate 
assumption ean also be interpreted as: 


D-i + S-i - ce 


(44) 


We will use this form later to find the optimal value of the parameter A. Combining (|4T]) and 


(43), using (34), and defining: 


N 




(45) 


2 = 1 


yields: 


AA^ + A - S ^ 0. 


(46) 


From (46), it is easy to see that A and S share the same eigenveetors. Denote the eigenvalue 
deeomposition of S by: 


S = U^diaglsj, i = 1, • • • , n}U' 


One eould then write A as: 


(47) 




A Usdiagjai, z = 1, • ■ ■ , n}U, 

To find Oj, z = 1, • • ■ , rz, we substitute ( [48] ) and ( [47] ) in ( |46| ) to obtain: 


(48) 


Aa^ + Oj — Sj ~ 0; z = 1, ■ ■ ■ , rz, 


whieh gives the following solution: 


\/l + 4Asj — 1 . 

“-; * = i, 


(49) 
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Note that the only unknown parameter in (49) is A. To find A, we use (44) and (38) to rewrite 


(42) as: 


Defining 7 as: 


N 

n 

2=1 




/ 3 . 


(50) 


N 


7 = 




-Cti 


2 = 1 


substituting (43) in (50), and simplifying the result yields: 


(51) 


A'^lA^i 


7- 


(52) 


Finally, substituting (49) in (52) leads to the following equation for A: 


j-j- f a/1 + 4Asj — 1 


2=1 


ViA 


7- 


(53) 


To summarize the results, given the parameters Wj, a*, Sn- for i = 1, ■ ■ ■ ,n, one should take 
the following steps to find the optimal alloeation of the distortions Dj to the nodes: 


Caleulate S using ( [45] ), and find Us and 57 i = 1, • ■ ■ , n from ( |47] ). 
Caleulate 7 from (jSTj) and then find A by solving (53). 


Use (49) and (p 8 |) to find A. 


Caleulate Zj using (43) 


Caleulate D* using (38). 


Note that the whole proeess depends on finding a solution for (53). Define Rmin as: 


Rrain 

We then prove the following proposition: 


nf=i 1 

a.S„,Wr^ 


-ott 1 


\Oti 


|S| 



Proposition 1. If R > Rmin, there is a unique solution to (p3]), otherwise (^) has no solution. 


Proof: See Appendix [^ 
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We perform a few simulations to further study the behaviour of the output SNR as a funetion 
of the distortion matriees Dj. We assume that there are two nodes N = 2, and Wi = W2 = In- 
We eonsider the following strueture for the eovarianee matrix of both signal and noise: 

where is the (?, j)th element of S. For the signal we ehoose u = 1 and p = 0.9. For 
the noise terms, we use u and p as eontrol parameters. The dimension and the weighted sum-rate 
are kept fixed on n = 32 and R = 2.5n, respeetively, unless otherwise stated. 

We eheek the optimality of the results and the gain that ean be aehieved. We first speeify the 
parameters ai, 02 , and the eovarianee matrix of the noise terms. We find the distortion matriees 
D*;z = 1,2 suggested by the proeedure deseribed above and their eorresponding SNR. We 
then eompare the resulting SNR to the SNR given by other ehoiees of the distortion matriees. 
Assuming that the eigenvalue deeomposition of D* is given by D* = UjAjUf, we use the 
following formula to generate other distortion matriees: 


Di = Vi {(3iAi + T]i&i) Vj, 


(54) 


where f3i and rji are weighting parameters, and ©j is a diagonal matrix whose diagonal elements 
are drawn from a uniform distribution in the interval [0,5^,], with Ci being the largest element 
of Aj. By ehoosing (3i elose to 1 and pi elose to 0, the resulting distortion matriees Dj will be 
slightly perturbed versions of D*. On the other hand, by ehoosing /3j elose to 0 and pi elose to 


1, the resulting Dj will be random. Using (54), we generate L valid distortion pairs (Di,D 2 ) 
using the following algorithm: 


Initialization: Set j = 1. 

Iterations: While j < L perform the following steps: 

1) Generate Di using ( [54| ) 

2) If the result is not a valid distortion matrix, go baek to 1, otherwise Generate D 2 using 

m. 


3) Normalize D 2 sueh that the eonstraint in ( [3^ holds. If the result is not a valid distortion 
matrix, go baek to 2. 
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400 600 

Trial 
(c) 


400 600 

Trial 
(d) 


Fig. 4. Output SNR for 1000 trials where the distortion matrices (Di,D 2 ) are slightly perturbed versions of (DTD 2 ). The 
blue line indicates the output SNR at (DTD 2 ). (a) with ai = 0:2 = 0.5, and pi = 0.9,p 2 = 0.3, ui = 0.01,1^2 = 0.02 for 
the covariance matrix of the noise terms, (b) ai = 02 = 0.5, and pi = P 2 = 0, vi = 0.01, V 2 = 0.02, (c) ai = 0.7, 02 = 0.3, 
and Pi = 0.9, p 2 = 0.3, vi = 0.01,122 = 0.02, (d) ai = 0.3, 0:2 = 0.7, and pi = 0.9, p 2 = 0.3, vi = 0.01,122 = 0.02. 


4) Calculate the SNR given by (Di,D 2 ). 

5) Inerease j by 1. 

We set L = 10^ for the rest of this subseetion. To study the behavior of the output SNR 
around (D^jD^), we set /3i = /52 = 0.999 and r/i = rj 2 = 0.001. Figure illustrates the results 
for different ehoiees of the parameters. As seen from Fig.Q the theoretieal results from the 
high-rate approximation indeed give the loeal maximum for the output SNR. 

To verify that (D^, D^) is the global maximizer of the output SNR and to see the gap between 
the maximum SNR and the one resulting from a random alloeation of the distortion matriees, 
we set /3i = /32 = 0 and rji = r ]2 = 1. The results are illustrated in Fig.[^ As seen from the 
plots, the gap between the optimal SNR and the one resulting from a random alloeation eould 
in some eases be as large as 5 dB. 

To show the sealability in sense of the number of sensors, we repeat the simulation in Fig. 

(a) with 4 sensors. Two sensors have similar parameters with sensor 1 in Fig.[^(a), and the 
other two sensors have similar parameters with sensor 2 in the same figure. The weights are 
set as «! = 02 = as = 04 = 0.25. The result is shown in Fig. whieh shows that for 4 
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Trial 

(c) 



0 200 400 600 800 1000 


Trial 

(4) 


Fig. 5. Output SNR for 1000 trials where the distortion matrices (Di,D 2 ) are chosen randomly. The blue line indicates the 
output SNR at (Di, D 2 ). (a) with ai = 02 = 0.5, and pi = 0.9, p 2 = 0.3, vi = 0.01, V 2 = 0.02 for the covariance matrix of 
the noise terms, (b) qi = a 2 = 0.5, and pi = p 2 = 0, v\ — 0.01, V 2 = 0.02, (c) ai = 0.7, a .2 = 0.3, and pi = 0.9, p 2 = 0.3, 
= 0.01, V 2 — 0.02, (d) Qi = 0.3, 02 = 0.7, and pi = 0.9, p 2 = 0.3, vi = 0.01, y 2 = 0.02. 

27 |- II I ^ 
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a. 

100 200 300 400 500 600 700 800 900 1000 
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Fig. 6 . Output SNR at 1000 trials for a simulation where there are 4 sensors and the distortion matrices (Di, D 2 , 03 , 04 ) are 
chosen randomly. The blue line indicates the output SNR at ( 01 , 02 , 03 , 04 ), which gives the optimal solution. The choice 
of parameters is similar to the simulation in Fig. 0 (a). 


sensors, as expected, the output SNR is higher compared to 2 sensors (Fig. (a)) due to the 
higher number of the available noisy signals. Moreover, comparison of the figures shows that 
the gap between the optimal performance and the one resulting from the random allocation of 
the distortion matrices is similar for the 4 sensors and 2 sensors scenarios. 

Next we study the validity of the high-rate assumption. Notice that even with low rates, the 
procedure described to find the optimal distortion allocation might give a solution. The problem. 
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Fig. 7. The difference between the desired weighted sum-rate R and the resulting weighted sum-rate due to the high-rate 
approximation as a function of R 


however is that if the rates are not high enough, the approximation in (43) will not hold, and the 


resulting distortion matrices might violate the sum-rate constraint in (33). The approximation 
error is negligible if aii?(D^) -f a 2 i?(D 2 ) ~ R. The difference between aii?(D^) 4- a 2 Rijy* 2 ) 
and i? as a function of R is plotted in Fig. |7] for cti = 02 = 0.5, and pi = p 2 = 0.8 and 
ui = 0.1, 1^2 = 0.2 for the covariance matrices of the noise terms. Note that for a weighted 
sum-rate R that is as low as 25 nats per block (equivalent to 0.78 nats per sample or 1.13 bits 
per sample) the high-rate assumption still holds with a negligible error. 


2) Scalar Sources: We assume that n = 1, which means that all the sources are scalar. We 
further assume that there are only two nodes. While the high-rate case studied in the previous 
subsection is of interest for practical applications, the scalar case with two nodes may be of 
less practical relevance. However, it leads to interesting analytical results that makes it worth 
studying. 

We denote the scalar version of D* and Wj by Di and Wi, respectively. The variance of 
random variables is denoted by S followed by a subscript. We assume that we are interested 
in the sum-rate (so oti = ot 2 = 0.5) and wi = W 2 to further simplify the problem for analytical 


derivations. Applying these assumptions to (38)-(42), and defining Si and S 2 as: 


Si — 


■‘rii 


■‘Vi 


7 = 1 , 2 , 


one can show that the target distortions Dl and D 2 are given by: 


DI = I3^ 


^712 


SniS,i2 ~ S2/3 

~ Si/3 J 


(55) 
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Fig. 8. An example of the output SNR as a function of D\ and D 2 for the scalar case with two nodes for three different 
values of the network sum-rate, (a) R < Rmin, (b) Rmin < R < Rmax, and (c) R > Rmax- The circles show the stationary 
points suggested by the KKT conditions. 




^n2 f^n\^n2 ~ 


(56) 


Based on this, and defining the following parameters: 


Rr, 


log 

[max(Ei,E2)] 

(57) 

-Ei)(S„,-S 2 )’ 

log 

[min(Ei,S2)]^ 

(58) 

-Ei)(E„,-E 2 )’ 


we prove the following proposition. 


Proposition 2. The stationary point given by {55)-{56) is the unique maximizer of the 

output SNR, if R TiYYLCLxy and is the unique minimizer, if Jd 

Proof: See Appendix ■ 

To observe the behaviour of the output SNR as a function of the distortions, consider the 
following example. Let wi = W 2 = 1, = 1, = 1.2, = 0.2, = 1.1, and 


= 0.1. From (57) and (58) we have Rmax = 1-13 and Rmin = 0.83. We make use of the 


weighted sum-rate constraint ^R{Di) + \R{D 2 ) = Rio write SNR(Zli, D 2 ) as a function of Di 
or D 2 only (denoted by SNR(Zli) and SNR(Zl 2 ), respectively). We plot SNR(Zli) and SNR(Zl 2 ) 
for three different values 0.5,1, 2 of the weighted sum-rate R. The result is shown in Fig.[^ As 
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seen in Fig.[^(a), for i? = 0.5 < Rmin the output SNR is minimized at {Di = D\,D 2 = D^), 
whieh is not desired. The maximum SNR should then be on the boundary, whieh means that the 
whole rate should be given to one of the nodes. It ean be seen in the figure that the highest SNR 
is achieved for the smallest D 2 , suggesting that the whole rate should be allocated to node 2. This 
is in agreement with intuition, since node 2 is less noisy than node 1. For Rmin < R < Rmax, 
one could see in Fig.[^(b) that {Di = Dl,D 2 = D 2 ) is not a feasible point. Similar to the 
previous case, the optimal solution should again be on the boundary. Finally, for R = 2 > Rmax, 
{Di = DI,D 2 = D 2 ) maximizes the output SNR. 

Note that the above example suggests that the distortion allocation has a water-filling form. 
The critical rate Rmax acts as the water level. If R is above this level, the rate is split between 
the nodes based on their noise levels. If R is below the water level, the noisier node is omitted, 
and the whole rate is given to the other node. 

IV. Conclusions and Discussion 

We considered a source coding problem in a networked setup under covariance matrix distor¬ 
tion constraints. We modelled the problem as a vector Gaussian remote Wyner-Ziv problem and 
solved the problem by deriving an explicit formula for the rate-distortion function and designing 
coding schemes that asymptotically achieve the rate-distortion function. We then studied some 
applications of the results. In particular, we showed that the rate-distortion function for the 
equivalent Wyner-Ziv problem with mean-squared error distortions and the rate-information 
function modelling a relay network source coding problem are special cases of our results. 
Finally, we considered a centralized sensor network with a weighted sum-rate constraint where 
each node transmits its observation with a certain rate and distortion, and the received data is 
fused at the center to maximize the output SNR without enforcing linear distortions. For this 
problem we bridged between noise reduction and source coding by showing that the distortion 
matrices and the rates at the individual nodes could be designed to maximize the output SNR at 
the center. We considered special cases such as the high-rate case or the case of scalar sources 
in order to obtain analytical results. Further work could be other possible special cases of the 
noise reduction problem where analytical results can be obtained. 
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Appendix A 

From the fact that n is uncorrelated with y and z and: 



it follows that: 


(59) 


(a b) = (e., e^) j 

Sxz^2^^zySy ^ SxzA 2^—SxyA^^SyzS^; ’ 
where Ai and A 2 are defined as: 

Al = Sy — SyzS^; ^^Zy, A2 = Sz ~ SzySy^Syz. 

Appendix B 
Proof of Lemma [3] 

We start from the following chain of inequalities: 


/?(D)=minJ(y;u z) s.t. Sxiuz ^ D,uOyf-)-(x, z) 

U 


>min/(y;u z) s.t. Sxiuz ^ D, uEA (y, z) <H-x 

U 

(60) 

= minJ(y;u z) s.t. Sxiuz ^ D, uO (y', z) Ox 

U 

(61) 

>min/(y'; u|z) s.t. Sxiuz ^D, uO (y', z) Ox 

U 

(62) 

1 Sxiuz ^ D, 

= min/i(y'z) —/i(y'u, z) s.t. < 

luo(y',z)ox 


1 ^y^lz 

> min -log-r- ^ s.t. < 

^y'|uz2 |Sy/|uz| 

< 

Sy'luz ^Sy/|z 

Sx|uz 

uo(y',z)ox 

(63) 
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> min -log- 

^y^|uz 2 


|Sy/|z| 


Sv/ 


y'|uz I 


S.t. Sy/|uZ Sy/jz, Sx|uZ D 


= min-log, 

^y'|uz2 


l^y'NI 


Sv' 


s.t. 


|Sy/|uz^Sx|z Sx|yz 


y'|uz| 


Sy'|uz ^D —S 


x|yz 


= ilog 


^y'|z I 


(64) 


(65) 


where ( [60| ) is beeause the knowledge of the side information z at the eneoder eannot inerease the 
rate, ( |M] ) is because from Q, when z is given, y' is a sufficient statistic of y for the estimation 
of X, follows from Q and the data processing inequality, (63) is because choosing y'|uz to 
be Gaussian maximizes the differential entropy, and we have added the constraint Sy/|uz ^ Sy/|z, 
since any valid Sy/|uz must satisfy this condition, (64) follows from (|^-(|^, and a* is defined 
as: 


* l-r-i \ ^ ) ^y'|uz ^ D Sx|yz, 

a = max|iiy/|uz| s.t. < 

y 1“^ I 5jy/|u2 ^x|z ^x|yz’ 


From Lemma it follows that a* = |min(D — Sx|yz, Sx|z — Sx|yz)|- Substituting this and ([^ 


in (65) yields: 


fi(D) > (log f;" 

^ |rnin(D Jj^iyz^^jxiz ^x|yz)| 

Appendix C 
Proof of Lemma |4] 

Based on the results in [|5l, flUl, it is enough to show that for u* defined in ( [TT| ), we have 
/(y;u*|z) = .R(D), and the covariance matrix Sx|u*z of the reconstruction error satisfies 
Sx|u*z ^ D- We start with the following chain of equalities: 


/ (y;u*|z) = h{u*\z) - h{u*\y,z) 


= ^log 
= ^log 



|USy,|,U'^ + S, 


( 66 ) 











34 


2 SJ 


(67) 


where (|66l) follows from ([TT]) and Q, and ([67]) follows from (|^. Rewriting A as 


A = U(S,|, - S,|yz)U' = UV~'AV-' U', 


( 68 ) 


and substituting (68) and (12) in (67) and simplifying the result, we get: 


7 (y;u*|z) 


^log| 

{ V-^AV-^ 

i V'^diag {min (Aj, A'), * = 1, ■ 

■ • • ,^x}V 

2log| 

1 51x|z ~ ^x|yz 1 


niin(D S^iyz? ^x|z ^x|yz) | 


Next, we derive the covariance matrix of the reconstruction error. Similar to (|^, we can write 
X as: 


X = Cu* + Gz + ni, 


(69) 


where rii is independent of u* and z. Form (69) it follows that Sxu*|z = CSu*| 2 , or: 




From ( [TT] ), ([^, and ([^ we have: 


(70) 


5ju*|z — a + Sjy. 


(71) 


Also note that: 


-'xu* |z 


= SxylzA^U^ 

= ASyi.A'^U^ 

= Sy'IzU^ 

= (Sx|z — Sx|yz) U , 


(72) 

(73) 

(74) 


( 75 ) 
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where ( |72l ), ( |7^ , ( [74] ) and ( |75| ) follow from ( [TT] ), Q, Q and Q, respeetively. The eovarianee 
matrix of the reeonstruetion error ean then be written as: 


Sx|u*z ^ni 

= 5jx|z ~ CSu*|zC^ 

= Sx|z — Sxu*|zS^.|2Sx^.|2 

= 5]x|z~ (Sx|z~5]x|yz)U'^(A + Sj^) U(Sx|z~5]x|yz) 
= Sxlyz + (Sx|z — Sx|yz) 

~ (Sx|z ~ 5]x|yz)U'^(A + Sj,) U (Sx|z ~ ^x|yz) 

= Sx|yz+V-lAV-^ 


- V-^AV-'^U^(A + S^)^^UV-^AV 


1tTA7-1 AA7-T 


= Sx|yz+V-lAV-^ 


—V ^A A+diag 


Ajmm(Ai,A') . 


= AV 


t-T 


V ' ‘"tAj-mm(Ai,A')' 

= Sx|yz + V“Miag {min (A*, A'), z = 1,..., n^c] 

^x|yz T min ^Sx|z ^x|yz) D Sx|yz) 


where ( |7^ and ( [77] ) follow from ( [69| ), ( [78] ) is the result of substituting ( |70| ) in ( |77| ), 
from ( |7T] ) and ( [75] ), ( [SO) ) follows from and ( [^ follows from ( [M] ) and ( [T2l ). 


Finally, rewriting (82) as: 


(76) 

(77) 

(78) 

(79) 


(80) 

(81) 

(82) 

follows 


^x|u*Z ^x|yz min ^Sx|z Sx|yZ 5 D Sx|yz) 5 


(83) 


and using Property 3 in Lemma it is elear that Sx|u*z ^ D, as desired. This eompletes the 
proof. 


Appendix D 

Derivation of the KKT Conditions 


We start with defining A as in Writing Dj in terms of Zj using ( [38] ), and substituting 
in the eonstraint in (|3^ yields (|42|). We derive (40) to eomplete the proof. Applying the matrix 
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inversion lemma to 


E„. + (Dr- - S-.') 


-1 


-1 


in (36), and using the definitions (38) and (41), 


we rewrite the optimization problem as follows: 


min tr{A 
Zi,--- ,Zjv 

N 


n + s-‘wrz-‘w,E-‘r“‘= p. 


(84) 


i=l 


For a given i G {I,-- - , N}, we differentiate the Lagrangian of (84) with respeet to Zj. To do 
so, we first apply a logarithm funetion to the eonstraint in ( [84| ), and then write the Lagrangian 
form of the problem as: 


N 


C{X, Zi, ■ ■ ■ , Z^)) = tr{(A, - Z,)“'} + a( /,(Z,)-log)g), 
where Aj and fjiTij) are defined as: 


,7 = 1 


(85) 


Aj — A + Zj, 

/i(Zi) = -ajlog 

Note that from (|^ it follows that A* does not depend on Zj. Using (39), we rewrite /j(Zj) as: 


+ S^WfZT^W.S^ 


/,(Z,) = - a, log (ZT^ - C-^) W,S-; 


= - aj log 




-1 


— ttj log C 


-11 


aj log IZ ■ M - aj log I Cj - Zj |. 


( 86 ) 


Substituting (86) in (85), differentiating with respeet to Zj while taking into aeeount that Zj is 
symmetrie [f2^ , and setting the derivative equal to 0 yields: 


2(Aj-Z,)-'-(A,-Z,)-'oI„ + 2Aaj[Z-' + (Q-Z,)-'] 
— Acij [Z- ^ + (Cj — Zj) ] o = 0, 


( 87 ) 


where o denotes the Hadamard produet. From (87) it follows that: 
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(A, - Z,)"' + Aai + (Q - Z,)-'] = 0 


( 88 ) 


Replacing Aj — Zj by A in ( 88 ) and simplifying the result yields (40). 


Appendix E 

Proof of Proposition [H 


Let us define: 


m ) = 


a/ 1 + 4Asi — 1 


;i = !,■■■ ,N. 


Using basic calculus one can show that //A) is a monotonically increasing and strictly concave 
function of A, which varies from 0 to Sj when A goes from 0 to cxd. It then follows that the 
function /(A) = Hii fiW — 7 is monotonically increasing with the range [— 7 , |S| — 7 [. One 
could then see that /(A) has one root, if and only if: 


|S|-7>0, (89) 

otherwise it has no root. Substituting ( [5T] ) in ( [89] ) and simplifying the result yields R > Rmin- 


Appendix F 
Proof of Proposition 

Applying the assumptions ai = 02 = 0.5, Wi = W 2 , and n = 1, the optimization problem is 
simplified to: 


max 


ni 


y 2 1 _I J_]_ 


1 

1 / 

^*12 

Sn2M 


S.t. D 1 D 2 = 


(90) 


^2 / 

Applying the following change of variables: 




( 91 ) 


and defining as: 
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= -S2)e 


AR 


(92) 


one could rewrite (90) as: 


mm 

Di,D' 


D[ + D', 


s.t. (d;-Si)(d'-s 2 ) = /3'. 


(93) 


Using the constraint in (93), we then write D 2 in terms of D[ and substitute it in the cost 


function to obtain an unconstrained function of D[. We call this function /i(i9(). Differentiating 
fi{D[) with respect to D[ yields: 


dfi{D[) 

dD[ 

(d;)'(S2u;-SiS2+/3')' 


(94) 


which is zero at D'^ = Si + (equivalent to Di = Dl using (91)). The sign of the derivative 


around this point determines whether there is an extremum or not. The denominator and the first 


term in the numerator in (94) are positive. We thus study only the second term in the numerator. 
We replace D[ by Si + + e. The result is {^/^ — S 2 )e, which implies that if ^/^ > S 2 , 

the derivative is positive for e > 0 and negative for e < 0. This means that if \/^ > ^ 2 , the 
minimum is at D[ = Si + or equivalently Di = Dl is the global maximizer of the SNR. 
Similarly, one could write D[ in terms of and substitute in the cost function to obtain f 2 {D 2 ). 
Considering the sign of the derivative around D 2 = leads to the conclusion that if > Si, 
the SNR is maximized at D 2 = D^. Combining the two conditions, the following should hold 
in order to have the SNR maximized at {Dl,Dl): 


> max(Si, S 2 ). 


(95) 


Substituting (92) in ([95]) yields: 


p. [max(Si,S2)]' 


= Rr. 


(96) 


Sl)(S„2 ~ ^ 2 ) 

Similarly one can show that for R < Rmin, (Rl, R 2 ) minimizes the output SNR. This 
completes the proof. 
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